test/e2e: make collectPodLogs log dump best-effort#6315
Conversation
The collectPodLogs helper runs from [AfterEach] to dump pod descriptions and container logs for the workload cluster. It currently uses Expect(...).To(Succeed()) when listing pods, which turns any transient inability to reach the workload cluster API server into a hard spec failure during teardown. This is the same bug pattern that was fixed for the sibling collectNodes helper in 4a8e9f1 (kubernetes-sigs#6265). With collectNodes now returning gracefully on i/o timeout, collectPodLogs is the next call in the AfterEach log collection path and hits the same transient unreachability against *.cloudapp.azure.com:6443, causing otherwise-successful runs of the apiversion-upgrade job to fail in [AfterEach]. Match the pattern already used a few lines below for streaming container logs (and now used in collectNodes): log the error and continue instead of failing the spec. Signed-off-by: Matt Boersma <Matt.Boersma@microsoft.com>
k8s-ci-robot
added
release-note
k8s-ci-robot
added
the
cncf-cla: yes
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #6315 +/- ##
=======================================
Coverage 43.84% 43.84%
=======================================
Files 289 289
Lines 25346 25346
=======================================
Hits 11114 11114
Misses 13458 13458
Partials 774 774 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
/cherry-pick release-1.23 |
|
@mboersma: once the present PR merges, I will cherry-pick it on top of DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/cherry-pick release-1.24 |
|
@mboersma: once the present PR merges, I will cherry-pick it on top of DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
k8s-ci-robot
added
the
lgtm
|
LGTM label has been added. DetailsGit tree hash: 255f03e310c28406dfb2b4f5ab11bb137bbbef6a |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: willie-yao The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
approved
|
@mboersma: new pull request created: #6317 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
@mboersma: new pull request created: #6318 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
4 participants
What type of PR is this?
/kind flake
What this PR does / why we need it:
Mirrors the fix from #6265 (commit 4a8e9f1) for the sibling
collectPodLogshelper.collectPodLogsruns from[AfterEach]to dump pod descriptions and container logs for the workload cluster, and currently usesExpect(...).To(Succeed())when listing pods. That turns any transient unreachability of the workload cluster API server into a hard spec failure during teardown.After #6265,
collectNodesreturns gracefully on i/o timeout, butcollectPodLogsis the very next call in theAfterEachlog-collection path and hits the same transient*.cloudapp.azure.com:6443: i/o timeoutagainst the same load balancer. The result is that otherwise-successful runs of theapiversion-upgrade-v1beta1job fail in[AfterEach]attest/e2e/azure_clusterproxy.go:112.Recent example: 5 of 6 consecutive failures on the cherry-pick PR #6314 show
STEP: PASSED!followed by[FAILED] in [AfterEach] - .../azure_clusterproxy.go:112, with the same i/o timeout signature against the management cluster LB in canadacentral.Match the pattern already used a few lines below for streaming container logs (and now used in
collectNodes): log the error and continue instead of failing the spec.Which issue(s) this PR fixes (optional, in
fixes #<issue number>(, fixes #<issue_number>, ...)format, will close the issue(s) when PR gets merged):Fixes #
Special notes for your reviewer:
This should be cherry-picked to release-1.24 so it can unblock #6314.
/cherry-pick release-1.24
TODOs:
Release note: